Bias Analysis in Entropy Estimation
نویسنده
چکیده
We consider the problem of finite sample corrections for entropy estimation. New estimates of the Shannon entropy are proposed and their systematic error (the bias) is computed analytically. We find that our results cover correction formulas of current entropy estimates recently discussed in literature. The trade-off between bias reduction and the increase of the corresponding statistical error is analyzed. PACS: 89.70+c, 02.50.Fz, 05.45.Tp Statistical fluctuations of small samples induce both statistical and systematic deviations of entropy estimates. In the naive (”likelihood”) estimator one replaces the discrete probabilities pi, for i = 1, ...,M , in the Shannon entropy [1]
منابع مشابه
A Note on Entropy Estimation
We compare an entropy estimator H(z) recently discussed by Zhang (2012) with two estimators, H(1) and H(2), introduced by Grassberger (2003) and Schürmann (2004). We prove the identity H(z) ≡ H(1), which has not been taken into account by Zhang (2012). Then we prove that the systematic error (bias) of H(1) is less than or equal to the bias of the ordinary likelihood (or plug-in) estimator of en...
متن کاملWEIGHTED k-NN GRAPHS FOR RÉNYI ENTROPY ESTIMATION IN HIGH DIMENSIONS
Rényi entropy is an information-theoretic measure of randomness which is fundamental to several applications. Several estimators of Rényi entropy based on k-nearest neighbor (kNN) based distances have been proposed in literature. For d-dimensional densities f , the variance of these Rényi entropy estimators of f decay as O(M), whereM is the sample size drawn from f . On the other hand, the bias...
متن کاملAssessing the Effects of Alzheimer’s disease on EEG Signals Using the Entropy Measure: a Meta-Analysis
Introduction and Aims: Alzheimer’s disease is the most prevalent neurodegenerative disorder and a type of dementia. 80% of dementia in older adults is because of Alzheimer’s disease. According to multiple research articles, Alzheimer's has several changes in EEG signals such as slowing of rhythms, reduction in complexity and reduction in functional associations, and disordered functional commun...
متن کاملOf fishes and birthdays: Efficient estimation of polymer configurational entropies
We present an algorithm to estimate the configurational entropy S of a polymer. The algorithm uses the statistics of coincidences among random samples of configurations and is related to the catch-tag-release method for estimation of population sizes, and to the classic “birthday paradox”. Bias in the entropy estimation is decreased by grouping configurations in nearly equiprobable partitions b...
متن کاملCorrecting sample selection bias in maximum entropy density estimation
We study the problem of maximum entropy density estimation in the presence of known sample selection bias. We propose three bias correction approaches. The first one takes advantage of unbiased sufficient statistics which can be obtained from biased samples. The second one estimates the biased distribution and then factors the bias out. The third one approximates the second by only using sample...
متن کامل